Interaction Control Based on Vision for AR Interface of Smart Phone

This paper proposes methods for detecting and tracking moving body parts such as the eyes, lips, and hands to enable natural interaction with smart phone platforms. For demonstrating these methods we have implemented a markerless-based augmented reality on an Android-based smart phone. For interaction the user does not use any extra appliances except for the natural movement of the eyes, lips and hands within the instantaneous field of view of the smart phone camera. To realize augmented reality, we developed state-of-the art computer vision algorithms to first detect and track the face using an Android API operating on the camera images, then we track the eye location within the face. We estimate the location of the lips rapidly by making inferences based on the information from the eye and face locations. Real-time detection of optimal locations of the movable bodily parts (such as hands) also use the smart phone camera. These methods allow the user to interact with virtual objects that are displayed on the smart phone. The demonstrated methods address current problems involving delayed detection time and unreliable tracking in markerless-based augmented reality applications. Our suggestions may serve to advance interaction control in smart phone augmented reality and serve as a platform for interface development.